Explorations & Insights

Large language model (LLM) reasoning is typically formulated as a conditional generation problem: given a question \mathbf{x} \in \mathcal{X}, a policy model \pi_{\theta}(\mathbf{y}|\mathbf{x}) generates an answer \mathbf{y} \in \mathcal{Y}. The quality of the answer is evaluated by a task-specific reward signal r(\mathbf{x}, \mathbf{y}). In reasoning tasks, the reward is usually sparse and terminal (e.g., correctness of the final answer), which means we consider one-step reward instead of returns (i.e., discounted sum of rewards over time steps).

This paper proposes DINO, a self-distillation framework with no labels, to pretrain ViTs. Besides the fact that the DINO method works quite well on this kind of architecture, there are also two interesting properties emerging from the learned features:

Currently, large models are post‑trained via RLHF, making them powerful but expensive to train and deploy, while smaller models are usually fine‑tuned with SFT or KD methods and are easier to deploy and adapt but often lack the performance of larger models.

令 f\in L^2(\mathbb{R})。傅里叶变换（在 L^2 意义下）把信号表示为全局正弦基的叠加：

In modern generative AI, encoders are commonly used during training to help models understand the context of input data. However, these encoders are often removed during inference. This raises an interesting question, if we train models using only decoders, can they still generate meaningful outputs?

The KL Divergence (KLD) is defined as

MuJoCo介绍